-
Notifications
You must be signed in to change notification settings - Fork 1k
feat: add validation for zombienet flaky tests #10034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
071ac7d
to
45c98c8
Compare
fd2bbc8
to
335530e
Compare
All GitHub workflows were cancelled due to failure one of the required jobs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about using gh
cli? Just to simplify the script a bit.
It is widely used in the CI.
eg.
gh issue comment $PR --edit-last -F msg.txt || gh issue comment $PR -F msg.txt |
Eg. we could get the issue state with this:
gh issue view 8944 --json state,title
Yes, I think is a better approach. We can just query the state and ensure is something like
thanks @DenzelPenzel / @lrubasze 🙌 |
f9f398d
to
a37c1ea
Compare
a37c1ea
to
150e8b3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Just noticed that PR numbers are treated like issue numbers.
See this:
❯ gh issue view 10034 --json state,title,url,id,stateReason,number
{
"id": "PR_kwDOKDT1Ss6t-Z9p",
"number": 10034,
"state": "OPEN",
"stateReason": "",
"title": "feat: add validation for zombienet flaky tests",
"url": "https://github.com/paritytech/polkadot-sdk/pull/10034"
}
vs
❯ gh issue view 9980 --json state,title,url,id,stateReason,number
{
"id": "I_kwDOKDT1Ss7QkIP6",
"number": 9980,
"state": "OPEN",
"stateReason": "",
"title": "Fix zombienet-polkadot-functional-0014-chunk-fetching-network-compatibility",
"url": "https://github.com/paritytech/polkadot-sdk/issues/9980"
}
Not sure. Maybe we could just ignore it. Alternatively we could detect it it by checking the url
or id
in order to distinguish if it is an issue or not.
I think test |
Yes, I think from GH point of view are two variant (issue/pr) and the number will not be repeated. We should add a regex to check the Thx! |
Description
Add some check in CI that would verify if test has a corresponding issue and if that issue exists. That verification could happen whenever some flaky test is added to the zombienet-flaky-tests.